Skip to content

Let evals boot canonical Agent Specs - #105

Merged
myobie merged 4 commits into
mainfrom
schickling-assistant/2026-07-30-eval-canonical-agent-spec
Jul 30, 2026
Merged

Let evals boot canonical Agent Specs#105
myobie merged 4 commits into
mainfrom
schickling-assistant/2026-07-30-eval-canonical-agent-spec

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes #104.

Why

The agent-new interview eval in compoundingtech/evals#56 needs to exercise the same account-neutral Agent Specs used by the real st2 catalog. The compact eval team grammar cannot represent that launch contract, and projecting it into wrappers creates a second authority.

What

  • add an explicit bare canonical-agents eval directive
  • recursively discover and strictly validate the hermetic catalog before spawn
  • preserve explicit Agent Spec identity and host authority independent of declaration placement
  • project only declarations resolved to the eval host; valid remote-host declarations remain inert
  • derive launch admission, supervision, PTY logging, and teardown from every admitted local runtime task instead of inventing a main PTY
  • require every resolved local task runtime ID to be nonempty and fleet-unique
  • freeze native inbox/archive routes at each admitted declaration's parent path
  • propagate backend boot errors and require every admitted launchable task to start
  • retain multi-agent completion ordering
  • for a singleton, snapshot requester entries before kickoff and accept only a newly appearing target reply at-or-after the exact kickoff timestamp
  • gate canonical verdicts on completion while preserving compact eval and flat-bus semantics without the opt-in

Evidence

  • cargo test --lib 'eval_' -- --nocapture (39 passed)
  • cargo test --test eval_run_e2e canonical_agents_ -- --nocapture (6 passed)
  • cargo test --test eval_run_e2e -- --nocapture (15 passed)
  • cargo test --workspace --all-targets -- --test-threads=1 (full workspace passed)
  • path-independent E2E admits a custom-named local PTY from an arbitrary declaration path, routes completion through native st2 message send, proves a valid remote declaration remains inert, and captures the custom task log
  • direct teardown evidence uses the admitted local Agent Specs with catch-all reap disabled, proves both a local PTY and local exec task are killed, and leaves an unrelated remote session untouched
  • singleton unit controls reject a future-dated pre-seeded reply and accept a causally new same-millisecond reply
  • exact-head Nix CI passed in 5m38s on run 30576505920 (job 90985998327) at 45210b29c1f874c189af8cae822d723a94f832d1
  • a temporary external-composition eval rendered its absolute hermetic workspace during a deterministic run, discovered one canonical account-neutral Agent Spec, materialized its persona/bus overlay, booted axe agent launch --harness claude ..., and routed the native kickoff
  • the captured PTY reached the real Claude workspace-trust screen: st2 admission, Axe trajectory admission, and real harness execution are proven; provider readiness and a paid model turn remain intentionally unproven because no automation accepted that human trust prompt
  • reproduced the original evals#56 pin failure at st2 9887b284: seat(s) ["new.interviewer"] exited at boot

Deliberately not done

No Agent Spec fields were duplicated in the eval grammar. No canonical path convention, main-PTY convention, provider/account wrapper, ambient-root compatibility bridge, generated declaration, or fallback to compact agents was added. The live Axe probe is external composition evidence, not a new st2-owned provider or trust-prompt contract.

agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f
@myobie
myobie marked this pull request as ready for review July 30, 2026 20:02

@myobie myobie left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent CoS review at exact head 45210b2: GREEN. Explicit identity/host authority, inert remote projection, path-derived native routes, unified PTY/exec runtime handling, fail-closed admission, causal completion, and scoped teardown match the accepted contract. Owner local matrices and exact-head hosted Nix are green.

@myobie
myobie merged commit c6846f6 into main Jul 30, 2026
1 check passed
@schickling schickling added area:agent-spec Agent Spec format, parsing, and the agent-spec crate · Set: manual area:catalog Catalog structure, declarations, transactions, and admission · Set: manual area:eval st2 eval harness and fixtures · Set: manual origin:agent Filed or primarily produced by an AI agent · Set: manual type:feature New user-visible or system capability · Set: manual labels Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent-spec Agent Spec format, parsing, and the agent-spec crate · Set: manual area:catalog Catalog structure, declarations, transactions, and admission · Set: manual area:eval st2 eval harness and fixtures · Set: manual origin:agent Filed or primarily produced by an AI agent · Set: manual type:feature New user-visible or system capability · Set: manual

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let eval teams boot canonical Agent Specs from the hermetic catalog

3 participants